feat: Image search example #5

Green-Sky · 2023-06-17T18:47:40Z

I used the usearch vector database library, to build a image embedding database and then to search in it by similarity using a text string.

build the database:

$ bin/image-search-build -m ../models/ggml-model-f16.bin ../tests/
clip_model_load: loading model from '../models/ggml-model-f16.bin' - please wait....................................................clip_model_load: model size =   288.93 MB / num tensors = 397
clip_model_load: model loadded
main: starting base dir scan of '../tests/'
main: found image file '../tests/white.jpg'
main: found image file '../tests/red_apple.jpg'

it generates an images.usearch and images.path file.

search by similarity to string:

$ bin/image-search -m ../models/ggml-model-f16.bin "a red apple"
clip_model_load: loading model from '../models/ggml-model-f16.bin' - please wait....................................................clip_model_load: model size =   288.93 MB / num tensors = 397
clip_model_load: model loadded
search results:
similarity path
  0.655244 /home/green/workspace/clip.cpp/tests/red_apple.jpg
  0.773273 /home/green/workspace/clip.cpp/tests/white.jpg

It is a very rough implementation. It requires c++17 and std::filesystem for iterating the filesystem.

A similarity search via an image would be nice too.

basically we could copy all the features from here: https://github.com/yurijmikhalevich/rclip

TODO:

add README.md to example
make it compile conditionally (bc c++17 and deps)
solve the usearch memory leak (tracking Bug: index.hpp leaks memory unum-cloud/usearch#138)
tabs to spaces
use fetch content instead of committing deps

monatis

I like the idea to provide an image search example with tiny vector search implementation directly in C++, but I highlighted some points.

Also, let's not use tab characters for indents please.

general code clean up. but the rest of the repo is still in a similar state 😝

Things may be messy at early stages to quickly introduce new features, but I'm trying to tidy up the code and README as we make progress in the project. You can always suggest better structures for the code and the overall project.

examples/image-search/usearch/index.hpp

examples/image-search/build.cpp

Green-Sky · 2023-06-18T10:08:32Z

Also, let's not use tab characters for indents please.

Ohhhhh yea. sorry. forgot to reverse this 😆

Green-Sky · 2023-06-18T10:13:49Z

Things may be messy at early stages to quickly introduce new features, but I'm trying to tidy up the code and README as we make progress in the project. You can always suggest better structures for the code and the overall project.

yea, I also intentionally maked this PR as draft for those reasons. :)

monatis · 2023-06-18T11:48:32Z

yea, I also intentionally maked this PR as draft for those reasons. :)

Great, thank you. Your work really appreciated. And to be honest, I was not aware of Usearch, but instead considering the original hnswlib library for the search example. But Usearch looks like a better candidate.

Green-Sky · 2023-06-18T15:51:35Z

forcepushed with FetchContent and spaces instead of tabs.

examples/image-search/search.cpp

monatis · 2023-07-03T07:06:42Z

Do you need an extra hand in this?

Green-Sky · 2023-07-03T22:59:30Z

Was doing something completely different last week :)
Will try to get this into a mergeable state this week.

(edit: why did you have to invert the cmake options <.<)

Green-Sky · 2023-07-07T17:11:22Z

just rebased and changed some stuff you committed on main

issues that I found:

you added cmake options for conditionally disabling tests and examples, i negated them and initialized them to standalone (more convenient)
you require c++20 (!) should only be 11 i think. (and 17 for fs in image-search until its changed to something else)
you set gcc/clang specific warning and O3 optimization level indiscriminately

clip.cpp/CMakeLists.txt

Line 136 in 1f140e0

add_compile_options(-Wno-format -O3)

(you should not, you default to Release

clip.cpp/CMakeLists.txt

Line 7 in 1f140e0

set(CMAKE_BUILD_TYPE Release CACHE STRING "Build type" FORCE)

, which sets O3)

monatis · 2023-07-07T17:29:00Z

Yes CMakeLists.txt still requires some testing and optimizations (also for other arches like different arm versions) so it might be another PR when I take down my Raspberry PIs from the shelf. C++20 standard was just a guard but we can downgrade it now or after some tests. I can refactor the directory walk with the function from common-clip that does not require C++17. Then everything will be ok with C++11.

I'm ok with it as is, and then we can continue to further improve it. When #30 is merged, we can also the use batch inference when indexing images.

Green-Sky · 2023-07-07T17:34:19Z

Yes CMakeLists.txt still requires some testing and optimizations (also for other arches like different arm versions) so it might be another PR when I take down my Raspberry PIs from the shelf. C++20 standard was just a guard but we can downgrade it now or after some tests. I can refactor the directory walk with the function from common-clip that does not require C++17. Then everything will be ok with C++11.

yea, there have also been changes upstream (especially llama) , where you obviously ~~copied~~ based the code on.

I'm ok with it as is, and then we can continue to further improve it. When #30 is merged, we can also the use batch inference when indexing images.

Yea. Let me run some small tests and then i will make it ready for review.

monatis · 2023-07-07T17:43:25Z

yea, there have also been changes upstream (especially llama) , where you obviously copied based the code on.

Exactly. The entire ggml-based projects are moving very fast, so I believe we don't need to target 100% production-grade multiplatform-ability. The speed of iterations and innovations is prioritized currently.

Yea. Let me run some small tests and then i will make it ready for review.

Great. Thank you.

monatis · 2023-07-08T06:02:19Z

USearch reports distance values instead of similarities, so I made changes to emphasize it. LGTM, merging. TODOs can be fixed later.

Thanks for your contributions.

Green-Sky force-pushed the image-search branch 2 times, most recently from 9553d20 to 77622a7 Compare June 17, 2023 18:55

monatis requested changes Jun 18, 2023

View reviewed changes

examples/image-search/usearch/index.hpp Outdated Show resolved Hide resolved

examples/image-search/build.cpp Outdated Show resolved Hide resolved

examples/image-search/build.cpp Outdated Show resolved Hide resolved

Green-Sky force-pushed the image-search branch from 77622a7 to b6d3c56 Compare June 18, 2023 15:49

Green-Sky force-pushed the image-search branch from b6d3c56 to 0a14626 Compare June 18, 2023 15:54

monatis reviewed Jun 18, 2023

View reviewed changes

examples/image-search/search.cpp Show resolved Hide resolved

Green-Sky force-pushed the image-search branch 5 times, most recently from 2385eaa to 7d170cf Compare June 25, 2023 13:50

Green-Sky force-pushed the image-search branch 3 times, most recently from 924cc29 to dfad626 Compare June 26, 2023 19:56

Green-Sky added 7 commits July 7, 2023 18:52

build embeddings database

0d1c5f0

add search

70ec88f

add verbosity and number of results cli args

ae23063

make examples and image-search be build conditionally

2c024f2

(edit: why did you have to invert the cmake options <.<)

save model path to image index and load from image index

b1ed0fe

fix api change from main

20bb778

cmake has no negation (except NOT in conditional expressions)

592147e

Green-Sky force-pushed the image-search branch from dfad626 to 592147e Compare July 7, 2023 17:03

Green-Sky mentioned this pull request Jul 7, 2023

Bug: index.hpp leaks memory unum-cloud/usearch#138

Closed

2 tasks

Green-Sky added 2 commits July 8, 2023 02:41

add readme

1eb480e

move vector out of loop

a1f34eb

Green-Sky marked this pull request as ready for review July 8, 2023 00:44

Green-Sky and others added 3 commits July 8, 2023 02:45

move image search reame

9b719c4

Fix some typos and notes in readme

e605bdd

similarity -> distance

181b44b

monatis merged commit 774c46a into monatis:main Jul 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Image search example #5

feat: Image search example #5

Green-Sky commented Jun 17, 2023 •

edited

Loading

monatis left a comment

Green-Sky commented Jun 18, 2023 •

edited

Loading

Green-Sky commented Jun 18, 2023

monatis commented Jun 18, 2023

Green-Sky commented Jun 18, 2023

monatis commented Jul 3, 2023

Green-Sky commented Jul 3, 2023

Green-Sky commented Jul 7, 2023

monatis commented Jul 7, 2023

Green-Sky commented Jul 7, 2023

monatis commented Jul 7, 2023

monatis commented Jul 8, 2023

feat: Image search example #5

feat: Image search example #5

Conversation

Green-Sky commented Jun 17, 2023 • edited Loading

TODO:

monatis left a comment

Choose a reason for hiding this comment

Green-Sky commented Jun 18, 2023 • edited Loading

Green-Sky commented Jun 18, 2023

monatis commented Jun 18, 2023

Green-Sky commented Jun 18, 2023

monatis commented Jul 3, 2023

Green-Sky commented Jul 3, 2023

Green-Sky commented Jul 7, 2023

monatis commented Jul 7, 2023

Green-Sky commented Jul 7, 2023

monatis commented Jul 7, 2023

monatis commented Jul 8, 2023

Green-Sky commented Jun 17, 2023 •

edited

Loading

Green-Sky commented Jun 18, 2023 •

edited

Loading